An Empirical Comparative Assessment of Inter-Rater Agreement of Binary Outcomes and Multiple Raters

Background: Many methods under the umbrella of inter-rater agreement (IRA) have been proposed to evaluate how well two or more medical experts agree on a set outcomes. The objective this work was assess key IRA statistics in context multiple raters with binary Methods: We simulated responses several (2–5) 20, 50, 300, and 500 observations. For each combination observations, we estimated expected value variance four commonly used (Fleiss’ Kappa, Light’s Conger’s Gwet’s AC1). Results: In case equal outcome prevalence (symmetric), values all were equal. asymmetric case, only three Kappa symmetric Fleiss’ yielded higher than other statistics. AC1 lower for scenario. Conclusion: Since population-level outcomes may not be known priori, statistic should favored over meaningful direct comparisons between measures, transformations conducted.